Virtual machine update while keeping devices attached to the virtual machine

ABSTRACT

An operation of a VM running first and second VM components is suspended so that a servicing operation for the VM can be performed. The VM has devices directly attached to it. A state of the first VM components is saved. An identification pointer for the second VM components is saved in a portion of the computing system physical memory without removing any underlying data structures of second VM components from computing system physical hardware. The directly attached devices remain configured as attached to the VM and remain configured to communicate with the VM while the VM is suspended and while the servicing operation is performed. The first VM components are shut down and then restored at the completion of the servicing operation using the saved state. The restored first VM components are reconnected to the second VM components using the identification pointers. The operation of the VM is restored.

BACKGROUND

The updating of host operating systems traditionally requires a rebootand therefore workload downtime. For virtualization hosts runningvirtual machine (VM) instances, downtime also implies downtime of theworkloads in hosted virtual machines. Virtualization technology oftenallows running VM contexts to be preserved by pausing execution of VMsand writing their associated RAM contexts to disk. However, suchoperations are typically relatively slow and result in detectableoutages, especially where the storage subsystem is slow or the amount ofmemory to be preserved and therefore the amount of IO required would bevery large.

In some update operations, rather than writing VM memory contents todisk, the host memory pages used for VM memory are catalogued such thatthe virtualization software can reassemble the state of VMs across theupdate operation. During the update operation, the current operatingsystem catalogues the VM artifacts, tears down the VM, prepares the newsoftware for execution, unloads the current software, and loads theupdated software which reads the meta-data and reconstructs the VMsusing the same memory pages as previously used. While this can be fasterthan writing memory contents to disk, the duration of the operation andtherefore the window of downtime is still negatively affected by severalfactors, including: 1) the quantity and continuity of memory to bepreserved 2) the destruction (including un-mapping) and re-construction(including re-mapping) of VM artifacts, especially the guest to hostpage mappings. In addition, VM instances which have assigned devicescannot participate in the update operation because assigned devicecannot be paused or quiesced across the operation nor can their state besaved and restored.

The subject matter claimed herein is not limited to embodiments thatsolve any disadvantages or that operate only in environments such asthose described above. Rather, this background is only provided toillustrate one exemplary technology area where some embodimentsdescribed herein may be practiced.

BRIEF SUMMARY

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter.

At least one embodiment disclosed herein is related to computing systemsand methods for performing a servicing operation on a virtual machine(VM). A computing system has first virtual machine components and secondVM components that remain loaded in computing system physical hardwareduring the servicing operation. An operation of a VM running the firstand second VM components is suspended so that the servicing operationfor the VM can be performed. The VM has devices that are directlyattached to it. A state of the first VM components is saved. Anidentification pointer for the second VM components is saved in aportion of the computing system physical memory without removing anyunderlying data structures of the second VM components from thecomputing system physical hardware. The directly attached devices remainconfigured as attached to the VM and configured to be in communicationwith the VM while the VM is suspended and while the servicing operationis performed. The first VM components are shut down and then restored atthe completion of the servicing operation using the saved state. Therestored first VM components are reconnected to the second VM componentsusing the identification pointers. The operation of the VM is restored.

Additional features and advantages will be set forth in the descriptionwhich follows, and in part will be obvious from the description, or maybe learned by the practice of the teachings herein. Features andadvantages of the invention may be realized and obtained by means of theinstruments and combinations particularly pointed out in the appendedclaims. Features of the present invention will become more fullyapparent from the following description and appended claims, or may belearned by the practice of the invention as set forth hereinafter.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to describe the manner in which the above-recited and otheradvantages and features can be obtained, a more particular descriptionof the subject matter briefly described above will be rendered byreference to specific embodiments which are illustrated in the appendeddrawings. Understanding that these drawings depict only typicalembodiments and are not therefore to be considered to be limiting inscope, embodiments will be described and explained with additionalspecificity and detail through the use of the accompanying drawings inwhich:

FIG. 1 illustrates an example computing system in which the principlesdescribed herein may be employed;

FIG. 2 illustrates an example host computing system for a VirtualMachine (VM);

FIGS. 3A-3C illustrate an example VM and environment for performing aservicing operation on the VM;

FIG. 4 illustrates a flowchart of an example method for performing aservicing operation on a VM.

DETAILED DESCRIPTION

The updating of host operating systems traditionally requires a rebootand therefore workload downtime. For virtualization hosts runningvirtual machine (VM) instances, downtime also implies downtime of theworkloads in hosted virtual machines. Virtualization technology oftenallows running VM contexts to be preserved by pausing execution of VMsand writing their associated RAM contexts to disk. However, suchoperations are typically relatively slow and result in detectableoutages, especially where the storage subsystem is slow or the amount ofmemory to be preserved and therefore the amount of IO required would bevery large.

In some update operations, rather than writing VM memory contents todisk, the host memory pages used for VM memory are catalogued such thatthe virtualization software can reassemble the state of VMs across theupdate operation. During the update operation, the current operatingsystem catalogues the VM artifacts, tears down the VM, prepares the newsoftware for execution, unloads the current software, and loads theupdated software which reads the meta-data and reconstructs the VMsusing the same memory pages as previously used. While this can be fasterthan writing memory contents to disk, the duration of the operation andtherefore the window of downtime is still negatively affected by severalfactors, including: 1) the quantity and continuity of memory to bepreserved 2) the destruction (including un-mapping) and re-construction(including re-mapping) of VM artifacts, especially the guest to hostpage mappings. In addition, VM instances which have assigned devicescannot participate in the update operation because assigned devicecannot be paused or quiesced across the operation nor can their state besaved and restored.

The embodiments disclosed herein provide improvements for updatingcomputing system running VMs over previous update methods.

At least one embodiment disclosed herein is related to computing systemsand methods for performing a servicing operation on a virtual machine(VM). A computing system has first virtual machine components and secondVM components that remain loaded in computing system physical hardwareduring the servicing operation. An operation of a VM running the firstand second VM components is suspended so that the servicing operationfor the VM can be performed. The VM has devices that are directlyattached to it. A state of the first VM components is saved. Anidentification pointer for the second VM components is saved in aportion of the computing system physical memory without removing anyunderlying data structures of the second VM components from thecomputing system physical hardware. The directly attached devices remainconfigured as attached to the VM and configured to re in communicationwith the VM while the VM is suspended and while the servicing operationis performed. The first VM components are shut down and then restored atthe completion of the servicing operation using the saved state. Therestored first VM components are reconnected to the second VM componentsusing the identification pointers. The operation of the VM is restored.

There are various technical effects and benefits that can be achieved byimplementing aspects of the disclosed embodiments. By way of example, atleast some of the embodiments the operation is not affected by thequantity and continuity of the guest to host memory mappings as will beexplained in more detail to follow. This is advantageously animprovement on current methods where the memory stack must be walked todetermine guest to host mappings. As will be appreciated, it may betimely and take computing system resources to walk the memory stack whenthere is a large number of discontinuous memory page mappings.

It is additionally an improvement on traditional methods as in thedisclosed embodiments some of the VM components have their underlyingdata structures remain loaded in the computing system physical resourcessuch as memory and processing resources. This saves on time as thesecomponents do not have to be rebuilt. In traditional methods, thesecomponents are destroyed and rebuilt, thus taking more time.

Further, the embodiments disclosed herein provide the technicalimprovement of allowing devices to remain directly attached to a VMduring an update process. This ability is not present in traditionalupdate methods.

Further, the technical effects related to the disclosed embodiments canalso include reduced power consumption, and can also include efficiencygains, as the quicker time will require less processing resources, whichcan in turn be used by other process of the computing system.

Some introductory discussion of a computing system will be describedwith respect to FIG. 1. Computing systems are now increasingly taking awide variety of forms. Computing systems may, for example, be handhelddevices, appliances, laptop computers, desktop computers, mainframes,distributed computing systems, datacenters, or even devices that havenot conventionally been considered a computing system, such as wearables(e.g., glasses). In this description and in the claims, the term“computing system” is defined broadly as including any device or system(or combination thereof) that includes at least one physical andtangible processor, and a physical and tangible memory capable of havingthereon computer-executable instructions that may be executed by aprocessor. The memory may take any form and may depend on the nature andform of the computing system. A computing system may be distributed overa network environment and may include multiple constituent computingsystems.

As illustrated in FIG. 1, in its most basic configuration, a computingsystem 100 typically includes at least one hardware processing unit 102and memory 104. The memory 104 may be physical system memory, which maybe volatile, non-volatile, or some combination of the two. The term“memory” may also be used herein to refer to non-volatile mass storagesuch as physical storage media. If the computing system is distributed,the processing, memory and/or storage capability may be distributed aswell.

The computing system 100 also has thereon multiple structures oftenreferred to as an “executable component”. For instance, the memory 104of the computing system 100 is illustrated as including executablecomponent 106. The term “executable component” is the name for astructure that is well understood to one of ordinary skill in the art inthe field of computing as being a structure that can be software,hardware, or a combination thereof. For instance, when implemented insoftware, one of ordinary skill in the art would understand that thestructure of an executable component may include software objects,routines, methods, and so forth, that may be executed on the computingsystem, whether such an executable component exists in the heap of acomputing system, or whether the executable component exists oncomputer-readable storage media.

In such a case, one of ordinary skill in the art will recognize that thestructure of the executable component exists on a computer-readablemedium such that, when interpreted by one or more processors of acomputing system (e.g., by a processor thread), the computing system iscaused to perform a function. Such structure may be computer-readabledirectly by the processors (as is the case if the executable componentwere binary). Alternatively, the structure may be structured to beinterpretable and/or compiled (whether in a single stage or in multiplestages) so as to generate such binary that is directly interpretable bythe processors. Such an understanding of example structures of anexecutable component is well within the understanding of one of ordinaryskill in the art of computing when using the term “executablecomponent”.

The term “executable component” is also well understood by one ofordinary skill as including structures that are implemented exclusivelyor near-exclusively in hardware, such as within a field programmablegate array (FPGA), an application specific integrated circuit (ASIC), orany other specialized circuit. Accordingly, the term “executablecomponent” is a term for a structure that is well understood by those ofordinary skill in the art of computing, whether implemented in software,hardware, or a combination. In this description, the terms “component”,“agent”, “manager”, “service”, “engine”, “module”, “virtual machine” orthe like may also be used. As used in this description and in the case,these terms (whether expressed with or without a modifying clause) arealso intended to be synonymous with the term “executable component”, andthus also have a structure that is well understood by those of ordinaryskill in the art of computing.

In the description that follows, embodiments are described withreference to acts that are performed by one or more computing systems.If such acts are implemented in software, one or more processors (of theassociated computing system that performs the act) direct the operationof the computing system in response to having executedcomputer-executable instructions that constitute an executablecomponent. For example, such computer-executable instructions may beembodied on one or more computer-readable media that form a computerprogram product. An example of such an operation involves themanipulation of data.

The computer-executable instructions (and the manipulated data) may bestored in the memory 104 of the computing system 100. Computing system100 may also contain communication channels 108 that allow the computingsystem 100 to communicate with other computing systems over, forexample, network 110.

While not all computing systems require a user interface, in someembodiments, the computing system 100 includes a user interface system112 for use in interfacing with a user. The user interface system 112may include output mechanisms 112A as well as input mechanisms 112B. Theprinciples described herein are not limited to the precise outputmechanisms 112A or input mechanisms 112B as such will depend on thenature of the device. However, output mechanisms 112A might include, forinstance, speakers, displays, tactile output, holograms and so forth.Examples of input mechanisms 112B might include, for instance,microphones, touchscreens, holograms, cameras, keyboards, mouse of otherpointer input, sensors of any type, and so forth.

Embodiments described herein may comprise or utilize a special purposeor general-purpose computing system including computer hardware, suchas, for example, one or more processors and system memory, as discussedin greater detail below. Embodiments described herein may also includephysical and other computer-readable media for carrying or storingcomputer-executable instructions and/or data structures. Suchcomputer-readable media can be any available media that can be accessedby a general purpose or special purpose computing system.Computer-readable media that store computer-executable instructions arephysical storage media. Computer-readable media that carrycomputer-executable instructions are transmission media. Thus, by way ofexample, and not limitation, embodiments of the invention can compriseat least two distinctly different kinds of computer-readable media:storage media and transmission media.

Computer-readable storage media includes RAM, ROM, EEPROM, CD-ROM orother optical disk storage, magnetic disk storage or other magneticstorage devices, or any other physical and tangible storage medium whichcan be used to store desired program code means in the form ofcomputer-executable instructions or data structures and which can beaccessed by a general purpose or special purpose computing system.

A “network” is defined as one or more data links that enable thetransport of electronic data between computing systems and/or modulesand/or other electronic devices. When information is transferred orprovided over a network or another communications connection (eitherhardwired, wireless, or a combination of hardwired or wireless) to acomputing system, the computing system properly views the connection asa transmission medium. Transmissions media can include a network and/ordata links which can be used to carry desired program code means in theform of computer-executable instructions or data structures and whichcan be accessed by a general purpose or special purpose computingsystem. Combinations of the above should also be included within thescope of computer-readable media.

Further, upon reaching various computing system components, program codemeans in the form of computer-executable instructions or data structurescan be transferred automatically from transmission media to storagemedia (or vice versa). For example, computer-executable instructions ordata structures received over a network or data link can be buffered inRAM within a network interface module (e.g., a “NIC”), and theneventually transferred to computing system RAM and/or to less volatilestorage media at a computing system. Thus, it should be understood thatstorage media can be included in computing system components that also(or even primarily) utilize transmission media.

Computer-executable instructions comprise, for example, instructions anddata which, when executed at a processor, cause a general purposecomputing system, special purpose computing system, or special purposeprocessing device to perform a certain function or group of functions.Alternatively or in addition, the computer-executable instructions mayconfigure the computing system to perform a certain function or group offunctions. The computer executable instructions may be, for example,binaries or even instructions that undergo some translation (such ascompilation) before direct execution by the processors, such asintermediate format instructions such as assembly language, or evensource code.

Although the subject matter has been described in language specific tostructural features and/or methodological acts, it is to be understoodthat the subject matter defined in the appended claims is notnecessarily limited to the described features or acts described above.Rather, the described features and acts are disclosed as example formsof implementing the claims.

Those skilled in the art will appreciate that the invention may bepracticed in network computing environments with many types of computingsystem configurations, including, personal computers, desktop computers,laptop computers, message processors, hand-held devices, multi-processorsystems, microprocessor-based or programmable consumer electronics,network PCs, minicomputers, mainframe computers, mobile telephones,PDAs, pagers, routers, switches, datacenters, wearables (such asglasses) and the like. The invention may also be practiced indistributed system environments where local and remote computingsystems, which are linked (either by hardwired data links, wireless datalinks, or by a combination of hardwired and wireless data links) througha network, both perform tasks. In a distributed system environment,program modules may be located in both local and remote memory storagedevices.

Those skilled in the art will also appreciate that the invention may bepracticed in a cloud computing environment. Cloud computing environmentsmay be distributed, although this is not required. When distributed,cloud computing environments may be distributed internationally withinan organization and/or have components possessed across multipleorganizations. In this description and the following claims, “cloudcomputing” is defined as a model for enabling on-demand network accessto a shared pool of configurable computing resources (e.g., networks,servers, storage, applications, and services). The definition of “cloudcomputing” is not limited to any of the other numerous advantages thatcan be obtained from such a model when properly deployed.

In embodiments, the computing system 100 may be implemented as a hostcomputing system that is capable of running one or more, and potentiallymany, virtual machines (VM). For instance, FIG. 2 abstractly illustratesa host 200 in further detail. In the case of FIG. 2, the host 200 isillustrated as operating three virtual machines 210 including virtualmachines 210A, 210B and 210C. However, the ellipses 210D once againrepresents that the principles described herein are not limited to thenumber of virtual machines running on the host 200. There may be as fewas zero virtual machines running on the host with the only upper limitbeing defined by the physical capabilities of the host 200.

During operation, the virtual machines emulate a fully operationalcomputing system including an at least an operating system, and perhapsone or more other applications as well. Each virtual machine is assignedto a particular client computer, and is responsible to support thedesktop environment for that client. As the user interacts with thedesktop at the client, the user inputs are transmitted from the clientto the virtual machine. The virtual machine processes the user inputsand, if appropriate, changes the desktop state. If such change indesktop state is to cause a change in the rendered desktop, then thevirtual machine alters the image or rendering instructions, ifappropriate, and transmits the altered image or rendered instructions tothe client computing system for appropriate rendering. From theprospective of the user, it is as though the client computing system isitself performing the desktop processing.

The host 200 includes a hypervisor 220 that emulates virtual resourcesfor the virtual machines 210 using physical resources 221 that areabstracted from view of the virtual machines 210. The hypervisor 221also provides proper isolation between the virtual machines 210. Thus,from the perspective of any given virtual machine, the hypervisor 220provides the illusion that the virtual machine is interfacing with aphysical resource, even though the virtual machine only interfaces withthe appearance (e.g., a virtual resource) of a physical resource, andnot with a physical resource directly. In FIG. 2, the physical resources221 are abstractly represented as including resources 221A through 221F.Examples of physical resources 221 include processing capacity, memory,disk space, network bandwidth, media drives, and so forth. The host 200may operate a host agent 202 that monitors the performance of the host,and performs other operations that manage the host. Furthermore, thehost 200 may include other components 203.

Attention is now given to FIGS. 3A-3C, which illustrate an embodiment ofa host 300, which may correspond to the host 200 previously described.The host 300 may include various components or functional blocks thatmay implement the various embodiments disclosed herein as will beexplained. The various components or functional blocks of host 300 maybe implemented on a local computing system or may be implemented on adistributed computing system that includes elements resident in thecloud or that implement aspects of cloud computing. The variouscomponents or functional blocks of the host 300 may be implemented assoftware, hardware, or a combination of software and hardware. The host300 may include more or less than the components illustrated in FIGS.3A-3C, and some of the components may be combined as circumstanceswarrant.

As illustrated, the host 300 may include or otherwise have access tophysical memory 305, which may correspond to memory 104 or one of thephysical resources 221 previously described. In the embodiments, thephysical memory 305 represents the physical memory resources, bothvolatile and non-volatile, of the host 300 that may be utilized by oneor more virtual machines as will be explained in more detail to follow.Accordingly, any reference to computing system physical memory in thisdisclosure or in the claims may refer to volatile RAM memory, persistentRAM memory, disk memory, storage memory, or any type of volatile andnon-volatile memory source. Thus, the embodiments and claims disclosedherein are not limited by the particular type of computing systemphysical memory that is implemented when practicing the embodimentsdisclosed herein.

The host 300 may also include or otherwise have access to a physicalprocessor 306, which may correspond to the physical processor 102 or oneof the physical resources 221 previously described. In the embodiments,the physical processor 306 represents the physical processing resourcesof the host 300 that may be utilized by one or more virtual machines aswill be explained in more detail to follow.

As illustrated, the host 300 may operate a virtual machine (VM) 310instance that may correspond to the one of the VMs 210 previouslydescribed. Although only one VM is shown as being operated by the host300, this is for ease of explanation only and accordingly theembodiments disclosed herein contemplate that the host 300 may operateany number of additional VMs as circumstances warrant. The host 300 mayalso include a host operating system (OS) 320 that may be able tocontrol the operation and function of the VM 310. The VM 310 and host OS320 will be explained in further detail to follow.

As mentioned previously in relation to FIG. 2, the host may include ahypervisor such as the hypervisor 220. Accordingly, the embodimentsdisclosed herein contemplate a hypervisor associated with the host 300and the VM 310. However, for ease of explanation, FIG. 3A does not showa hypervisor. Rather, FIG. 3A shows a hypervisor partition 315 that isgenerated by the hypervisor for the VM 310. As illustrated, thehypervisor partition 315 includes a first virtual processor 320 andsecond virtual processor 325. It will be noted that the hypervisorpartition 315 may include additional virtual processors as circumstanceswarrant. The virtual processors 220 and 225 are seen by the VM 310 asbeing actual processors. However, as illustrated by the dotted lines inthe physical processor 306, the virtual processors use the hypervisor toaccess the processing resources of the physical processor 306.

The hypervisor partition 315 also includes VM memory 330 withcorresponding Guest Page Address (GPA) space that is used to mapportions of the physical memory 305 to VM memory 330. For example, asillustrated in FIG. 3A a portion of physical memory 305A having anaddress range 307 and a portion of physical memory 305B having anaddress range 309 may be mapped by the hypervisor to portion 331 of VMmemory having an address range 331A. Likewise, a portion of physicalmemory 305C with an address range 308 may be mapped by the hypervisor toa portion 332 of the VM memory having an address range of 332A. In otherwords, the hypervisor is able to create and maintain a page table of themappings between the GPA space and the physical memory space.

The host OS 320 may include a VM worker process instance 340 that isinstantiated for running or configuring the VM 310. As illustrated, theVM worker process 340 may include a virtual device (Vdev) 345 and avirtual device (Vdev) 346. It will be noted that additional virtualdevices may be included as illustrated by the ellipses 347. Inembodiments, the virtual devices may be configured to emulate variousdevices associated with the VM 310. In embodiments, the VM workerprocess 340 may include additional components as also represented by theellipses 347. The additional components may include one or more of astate machine, a virtual motherboard, an IC proxy, or a RDP encoder. Inoperation, the VM worker process 340 may direct the hypervisor to createthe mapping between the VM memory 330 and the physical memory 305previously described.

In embodiments, the host OS 320 may include a driver 350. In oneembodiment, the driver 350 may be a virtual PCI driver that allows forthe control of one or more devices that are directly assigned orattached to the VM 310 such as a Graphical Processing Unit (GPU) 370and/or a Field Programmable Gate Array (FPGA) 380. The directly attacheddevices will be explained in more detail to follow. As illustrated bythe ellipses 351, additional drivers may also be included in the host OSas circumstances warrant. In embodiments, the driver 350 may be able to,independent of the hypervisor, map a portion 333 of VM memory 330 to aportion 305D of physical memory 305 for the use of the attached devices.In such embodiments, the driver 350 may maintain any page tables and thelike as needed.

As mentioned, the VM 310 may have various devices that are directlyassigned or attached to the VM. For example, one directly assigned orattached device may be the GPU 370. In some embodiments, the GPU 370 mayinclude an input-output memory management unit (IOMMU) 371 or becommunicatively coupled to an external IOMMU. Although not illustrated,in other embodiments the IOMMU 371 may be part of the physical processor306. The IOMMU 371 may have access to a corresponding address space 372of the GPU 370. In operation, the IOMMU 371 may interface with an attachmodule 316 of the hypervisor partition 315. The attach module 316represents the various functionality of the hypervisor that allows theGPU 370 to directly attach to the VM 310. The IOMMU 371 is then able todirectly access the GPA space of the VM memory 330 so that a mappingbetween the GPA space and the device address space may be made, whichallows the GPU 370 to directly access the physical memory 305 via the VMmemory 305. The GPU 370 may thus have access to the memory pages mappedto the VM memory 305.

In addition, the attach module 316 may allow the GPU 370 to directlyaccess the virtual processors 320 and 325 for needed processingresources. Said another way, the GPU 370 views at least one of thevirtual processors 320 and 325 as its assigned processor.

In like manner, another directly assigned or attached device may be theFPGA 380. As illustrated, the FPGA 380 may interface with the attachmodule 316. As illustrated, in some embodiments, the FPGA 380 mayinclude or otherwise have access to an IOMMU 381. Although notillustrated, in other embodiments the IOMMU 381 may be part of thephysical processor 306. The IOMMU 381 may perform a mapping between anaddress space 382 of the FPGA 380 and the GPA space of the VM memory 330in the manner previously described for the IOMMU 371. In addition, theFPGA 380 may directly access the virtual processors 320 and 325 forneeded processing resources.

The ellipses 383 represent that any number of additional devices may bedirectly assigned or attached to the VM 310. Examples of the additionaldevices 383 include, but are not limited to, Universal Serial Bus (USB)including USB3 controllers, storage controllers, Peripheral ComponentInterconnect Express (PCIe) devices, and Non-Volatile Memory Express(NVMe) storage devices. The additional devices 383 may also includevarious network cards and the like that expose registers at locations inthe computer's physical memory space that may be mapped to the GPA spaceof the VM memory 330. Accordingly, the embodiments disclosed herein arenot limited by the number or type of devices that are directly assignedor attached to the VM 310. It will be noted that for ease ofexplanation, the embodiments disclosed herein are described having theGPU 370 or the FPGA 380 directly attached to the VM 310. Accordingly,any discussion relating to the operation and the like of the GPU 370 andthe FPGA 380 will also apply to any of the other directly assigneddevices 383.

The host OS 320 may also include a management module 360. It will benoted that the management module 360 is used for ease of explanation andmay represent kernel mode processes such as a VirtualizationInfrastructure Driver (VID) and user mode processes such as VirtualMachine Management Service (VMMS). Accordingly, the embodimentsdisclosed herein are not limited by the actual type of the managementmodule 360.

In operation, the management module 360 may include storage resourcesthat are backed by portions the physical memory 305, such as the portion305E. In embodiments, the storage resources may be a device extension orthe like of the management module 360. In embodiments, the managementmodule may include a partition for each VM of the host 300. However, thememory resources (i.e., the device extension) may remain even if thepartitions are removed as long as the memory management module remainsloaded in the host 300.

In some embodiments, however, it may be possible to save the storageresources in a portion of the physical memory that has been designatedto be persisted even during a host OS 320 shut down. In suchembodiments, the management module 360 may be able to be unloaded whilestill persisting any information saved in the storage resources.

As shown in FIG. 3A, in embodiments the VM worker process 340 may directthe management module 360 to store various information in the memoryresources of the management module 360 for the partition correspondingto the VM 310. For example, the VM worker process 340 may direct themanagement module 360 to store a data block 361 that corresponds to thearray of memory pages mapped to the portion 331 of VM memory 330 and tostore the corresponding GPA address range 331A as shown at 362.Likewise, the VM worker process 340 may direct the management module 360to store a data block 363 that corresponds to the array of memory pagesmapped to the portion 332 of VM memory 330 and to store thecorresponding GPA address range 332A as shown at 364.

The VM worker process 340 may also direct the management module 360 togenerate a state file 365 that records the current state of the VMworker process 340. The current state of the various virtual devices 345and 346 may also be recorded in the state file 365. This file may beused to persist the state of the VM worker process and the virtualdevices as will be explained in more detail to follow.

In embodiments, the driver 350 may also direct the management module 360to record information. For example, the driver 350 may direct themanagement module 360 to record a data block 366 that corresponds to thearray of memory pages mapped to the portion 333 of VM memory 330. Thecorresponding GPA range may also be recorded. It will be noted that anyof the additional drivers 351 may also direct the management module 360to record information as circumstances warrant. Accordingly, theembodiments disclosed herein contemplate both the VM worker process 340and various drivers directing the management module 360 to recordinformation as needed.

In embodiments, it may be desirable to service one or more of thecomponents of the VM 310 and/or the host OS 320. The servicing may allowfor software updates and the like to happen to the relevant components.In the embodiments disclosed herein, the servicing operation may occurwithout the need to reboot the host OS 320. In addition, some of thecomponents will not have their underlying data structures removed fromthe computing system physical hardware (i.e., the physical memory 305and the physical processor 306) during the servicing operation. Thus,these components will typically not be subject to any update during theservicing operation.

In embodiments, the VM worker process 340 may initiate the VM servicingoperation by directing the management module 320 to store the variousinformation in its storage resources (i.e., device extension) in themanner previously described. For example, the management module 320 maysave the data blocks 361 and 363 and the corresponding GPA ranges 362and 364. In addition, the driver 350 may direct the management module320 to store the data block 366 and corresponding GPA range.

In response to storing the information, the management module 320 mayalso generate a store 390 for storing persistent identity pointers forcomponents of the VM that are not to have the servicing operationperformed on them, but that are to have their underlying data structuresremain loaded on the computing system physical hardware. For example, anidentity pointer 391 for the hypervisor partition 315 including thevirtual processors may be stored in the store 390. In addition, identitypointers 392-396 for the memory block 361, the GPA range 362, the memoryblock 363, the GPA range 364, and the memory block 366 respectively maybe stored in the store 390. The store 390 may then be written into adedicated portion 305F of the physical memory 305. It will be noted thatthe identity pointers 391-396 may be considered artifacts that point tothe underlying data structures in the computing system physical hardware(i.e., the physical memory 305 and the physical processor 306) for thecomponents corresponding to the identity pointers.

It will be noted that, in at least some implementations, only the datablocks in the management module 360 are persisted. In theseimplementations, there is no need to access the physical memory 305 todetermine the addresses of the data blocks to persist. This may resultin a time savings if there is a large number of data blocks to persistor if the data blocks are non-contiguous with each other. In otherwords, there is no need to walk the memory stack to determine the datablocks to persist.

The state of the VM worker process and virtual devices may also bestored in the state file 365. This state may then be stored in thememory resources of the management module 360 or in the portion 305E ofthe physical memory 305.

The operation of the VM 310 may then be suspended so that the servicingoperation may occur. FIG. 3B represents the VM 310 during the servicingoperation. For ease of illustration, some of the elements shown in FIG.3A are not included in FIG. 3B.

As shown in FIG. 3B, VM worker process 340 and the virtual devices 345and 346 have been shut down and thus are not present in the figure. Inother words, the VM worker process 340 and the virtual devices 345 and346 have been removed so that any of their underlying data structureshave been removed from the computing system physical hardware. Asmentioned previously, however, the state of VM worker process 340 andthe virtual devices 345 and 346 prior to being shut down was saved inthe state file 360 and may be used to restore these components as willbe discussed in more detail to follow.

In FIG. 3B, the hypervisor partition 315 and its components includingthe virtual processors 320 and 325 and the virtual memory 330 and itsassociated mappings with the physical memory 305 (i.e., 331 and 332) areshown as being dotted. The mapping 333 of the driver 350 is also shownas being dotted. The dots represent that although the operation of theVM 310 has been suspended, the underlying data structures for thesecomponents have not been removed from the computing system physicalhardware. In other words, even while the operation of the VM 310 hasbeen suspended and the servicing operation is being performed, thehypervisor partition 315 and the GPA mapping remain loaded or programedin the computing system physical hardware. It will be noted, however,that the operation of the components of the hypervisor partition issuspended while the operation of the VM 310 is suspended.

It will also be noted that the host OS 320 is not shut down during theservicing operation, but also remains loaded along with the managementmodule 360, although the specific partition in the memory managementmodule for the VM 310 has been removed. Thus, those components that arenot shut down are typically not able to be serviced by the servicingoperation as this typically requires that a component be shut down sothat any changes to the components during the servicing operation may beimplemented.

FIG. 3B also shows that the devices that are directly attached to the VM310 including the GPU 370 and the FPGA 380 remain attached or are keptactive while the operation of the VM 310 has been suspended. That is,from the perspective of the directly attached devices that are keptactive, the VM 310 is still in active operation and the devices and theVM 310 are able to normally function in relationship with each other.Thus, the attach module 316 allows the mapping between the GPA space ofthe VM memory 330 and the IOMMU previously described to remain intact.Since the GPA mapping has not been removed as previously described, themapping between the device and the VM 310 is able to be maintained.

The devices that are directly attached to the VM 310 including the GPU370 and the FPGA 380 also retain access to processing resources of thevirtual processors 320 and 325. As may be appreciated, since the devicesthat are directly attached to the VM 310 including the GPU 370 and theFPGA 380 may continue to operate while the operation of the VM 310 issuspended, there may be instances where the directly attached deviceinitiates an interrupt to be handled by one of the virtual processors.For example, the GPU 370 may initiate an interrupt 318 and the FPGA 380may initiate an interrupt 319. To handle such instances, the attachmodule or some other component of the VM 310 may include a queue file317 that is able to queue the interrupts while the virtual processorsare suspended. For example, in embodiments where the directly attacheddevice uses Message Signal Interrupts (MSI), the device has at most 32interrupts and an address that the device writes to indicate a pendinginterrupt. Where that address is mapped to the VM, the directly attacheddevice is allowed to write to that address, even when the operation ofthe VM is suspended. The same is true for MSI-X, except that MSI-X hasup to 2048 interrupts.

As will be explained in more detail, once the operation of the virtualprocessors is restored, the virtual processors are able to handle theinterrupts in the queue file 317. Thus, from the perspective of thedirectly attached devices this is only seen as a small delay inprocessing time.

FIG. 3C shows the process of restoring the VM 310 to operation. Asshown, FIG. 3C includes the host OS 320 and the store 390. After theservicing operation of the VM 310 is completed, which may result in oneor more components of the VM or host OS being updated or the like, theVM worker process 340 and the virtual devices 245 and 246 may berestored by the host computing system. As previously described, thestate of the VM worker process 340 and the virtual devices 245 and 246was saved before the operation of the VM 310 was suspended. This statemay then be used to reload or program the underlying data structures inthe computing system hardware to thereby restore the VM worker process340 and the virtual devices 245 and 246.

Once the VM worker process 340 is restored, the VM worker process maydirect the management module 360 to rebuild the partition for the VM 310and to build a data object for memory blocks and corresponding GPAranges and may also request that a hypervisor partition be generated. Insuch case, the management module 360 as shown at 302 may access theidentity pointers stored in the store 390. The identity pointers maythen allow the management module 360 to access the underlying datastructures for the various components stored in the store 390. Themanagement module 360 may then populate the data objects 361-364 and 366with the recovered underlying data structures as shown at 303. This hasthe effect of reconnecting the hypervisor partition 315, virtualprocessors 320 and 325, and the mapping of the virtual memory 330 to therestored VM worker process 340. That is, the VM worker process 340 isnow able to use these components without the need for the system toactually rebuild these components in the computing system physicalhardware. Said another way, since the hypervisor partition 315, virtualprocessors 320 and 325, and the mapping of the virtual memory 330 to thephysical memory 305 was not removed during the servicing operation, themanagement module 360 only needs to reconnect the VM worker process 340to these components.

The reconnecting of the VM worker process to the hypervisor partition315, virtual processors 320 and 325, and the mapping of the virtualmemory 330 to the physical memory 305 may return the VM to the stateshown in FIG. 3A prior to the generation of the store 390. That is, theVM 310 may resume its normal operation.

Upon the resumption of the operation of the VM 310, any interrupts suchas interrupts 318 and 319 stored in the queue file 317 may be handled bythe virtual processors 320 and 325. The directly attached devices suchas the GPU 370 and the FPGA 380 may then continue to normallycommunicate with and operate in relation to the resumed VM 310.

The following discussion now refers to a number of methods and methodacts that may be performed. Although the method acts may be discussed ina certain order or illustrated in a flow chart as occurring in aparticular order, no particular ordering is required unless specificallystated, or required because an act is dependent on another act beingcompleted prior to the act being performed.

FIG. 4 illustrates a flow chart of an example computerized method 400for servicing components of a VM while second components remain loadedin the computing system physical hardware during the servicingoperation. The method 400 will be described with respect to one or moreof FIGS. 2-3C discussed previously.

The method 400 includes suspending an operation of a VM running one ormore first and second VM components so that a servicing operation forthe VM may be performed (410). In some embodiment the VM may have one ormore devices that are directly attached to it. For example, aspreviously described the VM 310 may run or have associated with it theVM worker process 340 and the virtual devices 345 and 346, which may beexamples of a first VM component. In addition, the VM 310 may run orhave associated with it the hypervisor partition 315, virtual processors320 and 325, and the virtual memory 330 that has a GPA space that ismapped to the physical memory 305. These may be considered as examplesof a second VM component.

As previously described in some embodiments the VM 310 may have one ormore devices that are directly attached to it. For example, the GPU 370and the FPGA 380 may be directly attached to the VM 310.

The method 400 includes saving a state of the one or more first VMcomponents (420). For example, as previously described the state of theVM worker process 340 and the virtual devices 345 and 346 may be storedin the state file 365.

The method 400 includes saving an identification pointer for the one ormore second VM components in a portion of the computing system physicalmemory without removing any underlying data structures of the one ormore second VM components from the computing system physical hardware(430). For example, as previously described the identification pointers391-396 for the hypervisor partition, the memory blocks 361 and 363, andGPA ranges 362 and 364, as well as the memory block 366 may be stored inthe store 390, which may correspond to the portion 305F of the physicalmemory 305.

As previously described, the one or more directly attached devicesremain attached to the VM and remain configured to communicate with theVM while the VM is suspended and while the servicing operation isperformed since the underlying data structures of the one or more secondVM components are not removed. For example, the GPU 370 and the FPGA 380remain directly attached to and remain configured to communicate withthe VM 310 while the operation of the VM 310 is suspended. This happensbecause the underlying data structures of the hypervisor partition 315,virtual processors 320 and 325, and the virtual memory 330 that has aGPA space that is mapped to the physical memory 305 remain in thephysical hardware of the computing system while the VM is suspended.

The method 400 includes shutting down the one or more first VMcomponents by removing any underlying data structures for the one ormore first VM components from the computing system physical hardware(440). As previously described, the VM work process 240 and the virtualdevices 245 and 246 are shut down by having their underlying datastructures removed from the physical hardware of the computing system.

The method 400 includes restoring at the completion of the servicingoperation the one or more first VM components (450). For example, aspreviously described the underlying data structures in the computingsystem physical hardware of VM work process 240 and the virtual devices245 and 246 are rebuilt using the saved state from the state file 265.

The method 400 may include reconnecting the restored one or more firstVM components to the one or more second VM components using theidentification pointers (460). For example, as previously described theidentity pointers 391-396 may be used to access the underlying datastructures of the hypervisor partition 315, virtual processors 320 and325, and the virtual memory 330 that has a GPA space that is mapped tothe physical memory 305. These components may then be reconnected to theVM work process 240 and the virtual devices 245 and 246 in the mannerpreviously described.

The method 400 includes resuming the operation of the VM (470). Forexample, the operation of the VM 310 may be resumed in the mannerpreviously described.

For the processes and methods disclosed herein, the operations performedin the processes and methods may be implemented in differing order.Furthermore, the outlined operations are only provided as examples, andsome of the operations may be optional, combined into fewer steps andoperations, supplemented with further operations, or expanded intoadditional operations without detracting from the essence of thedisclosed embodiments.

The present invention may be embodied in other specific forms withoutdeparting from its spirit or characteristics. The described embodimentsare to be considered in all respects only as illustrative and notrestrictive. The scope of the invention is, therefore, indicated by theappended claims rather than by the foregoing description. All changeswhich come within the meaning and range of equivalency of the claims areto be embraced within their scope.

What is claimed is:
 1. A computing system having one or more firstvirtual machine (VM) components and one or more second VM componentsthat are to remain loaded in computing system physical hardware during aservicing operation for the VM, the computing system comprising: atleast one processor; a computer readable hardware storage device havingstored thereon computer-executable instructions which, when executed bythe at least one processor, cause the computing system to perform thefollowing: suspend an operation of a VM running the one or more firstand second VM components so that a servicing operation for the VM can beperformed, the VM having one or more devices that are directly attachedto it; save a state of the one or more first VM components; save anidentification pointer for the one or more second VM components in aportion of the computing system physical memory without removing anyunderlying data structures of the one or more second VM components fromthe computing system physical hardware, wherein the one or more directlyattached devices remain configured as attached to the VM and remainconfigured to communicate with the VM while the VM is suspended andwhile the servicing operation is performed since the underlying datastructures of the one or more second VM components are not removed; shutdown the one or more first VM components by removing any underlying datastructures for the one or more first VM components from the computingsystem physical hardware; restore at the completion of the servicingoperation the one or more first VM components; reconnect the restoredone or more first VM components to the one or more second VM componentsusing the identification pointers; and resume the operation of the VM.2. The computing system of claim 1, wherein the one or more first VMcomponents is one of a virtual device and a virtual machine workerprocess.
 3. The computing system according to claim 1, wherein the oneor more second VM components are one or more of a hypervisor partitionand corresponding virtual processor, a page mapping between the physicalcomputing system memory and virtual memory of the VM, or a virtualdevice driver configured to interface with the directly attached device.4. The computing system of claim 1, wherein the one or more second VMcomponents includes a virtual processor, the computing system beingfurther configured to perform the following: queue any interruptsgenerated by the one or more directly attached devices while the VM issuspended; and have the virtual processor handle the interrupts once theoperation of the VM is resumed.
 5. The computing system of claim 1,wherein the one or more second VM components includes a page mappingbetween the physical computing system memory and virtual memory of theVM, wherein the one or more directly attached devices use the mappingwhile in operation while the VM is suspended.
 6. The computing system ofclaim 1, wherein the one or more directly attached devices is a one ormore of a Graphical Processing Unit (GPU), a Field Programmable GateArray (FPGA), USB3 controllers, storage controllers, NVMe storagedevices. PCIe device, or a network card.
 7. The computing system ofclaim 1, wherein the second VM components are not subjected to theservicing operation.
 8. A computing system having one or more firstvirtual machine (VM) components that are to be subjected to a servicingoperation and one or more second VM components that are to remain loadedin physical computing system hardware during the servicing operation,the computing system comprising: at least one processor; a computerreadable hardware storage device having stored thereoncomputer-executable instructions which, when executed by the at leastone processor, cause the computing system to perform the following:suspend an operation of a VM running the one or more first and second VMcomponents so that a servicing operation for the VM can be performed;save a state of the one or more first VM components; save anidentification pointer for the one or more second VM components in aportion of the computing system physical memory without removing anyunderlying data structures of the one or more second VM components fromthe computing system physical hardware; shut down the one or more firstVM components by removing any underlying data structures for the one ormore first VM components from the computing system physical hardware;restore at the completion of the servicing operation the one or morefirst VM components by; reconnect the restored one or more first VMcomponents to the one or more second VM components using theidentification pointers; and resume the operation of the VM.
 9. Thecomputing system of claim 8, wherein the one or more first VM componentsis one of a virtual device and a virtual machine worker process.
 10. Thecomputing system of claim 8, wherein the one or more second VMcomponents is a hypervisor partition and corresponding virtualprocessor.
 11. The computing system of claim 8, wherein the one or moresecond VM components is a page mapping between the physical computingsystem memory and virtual memory of the VM.
 12. The computing system ofclaim 8, wherein the one or more second VM components is a listing ofblocks of the physical computing memory and their correspondingaddresses ranges in the physical computing memory that are used by thefirst and second VM components.
 13. The computing system of claim 8,wherein a directly attached device is attached to the one or more VMs,the directly attached device remaining attached to the one or more VMswhile the VMs are suspended because the underlying data structures ofthe one or more second VM components are not removed.
 14. The computingsystem of claim 8, wherein the second VM components are not subjected tothe servicing operation.
 15. In a computing system having one or morefirst virtual machine (VM) components and one or more second VMcomponents that are to remain loaded in computing system physicalhardware during a servicing operation of the VM, a method for performingthe servicing operation of the VM, the method comprising: suspending anoperation of a VM running the one or more first and second VM componentsso that a servicing operation for the VM can be performed, the VM havingone or more devices that are directly attached to it; saving a state ofthe one or more first VM components; saving an identification pointerfor the one or more second VM components in a portion of the computingsystem physical memory without removing any underlying data structuresof the one or more second VM components from the computing systemphysical hardware, wherein the one or more directly attached devicesremain configured as attached to the VM and remain configured tocommunicate with the VM while the VM is suspended and while theservicing operation is performed since the underlying data structures ofthe one or more second VM components are not removed; shutting down theone or more first VM components by removing any underlying datastructures for the one or more first VM components from the computingsystem physical hardware; restoring at the completion of the servicingoperation the one or more first VM components; reconnecting the restoredone or more first VM components to the one or more second VM componentsusing the identification pointers; and resuming the operation of the VM.16. The method of claim 15, wherein the one or more first VM componentsis one of a virtual device and a virtual machine worker process.
 17. Themethod of claim 15, wherein the one or more second VM components are oneor more of a hypervisor partition and corresponding virtual processor, apage mapping between the physical computing system memory and virtualmemory of the VM, or a virtual device driver configured to interfacewith the directly attached device.
 18. The method of claim 15, whereinthe one or more second VM components includes a virtual processor, thecomputing system being further configured to perform the following:queue any interrupts generated by the one or more directly attacheddevices while the VM is suspended; and have the virtual processor handlethe interrupts once the operation of the VM is resumed.
 19. The methodof claim 15, wherein the one or more second VM components includes apage mapping between the physical computing system memory and virtualmemory of the VM, wherein the one or more directly attached devices usethe mapping while in operation while the VM is suspended.
 20. The methodof claim 15, wherein the one or more directly attached devices is a oneor more of a Graphical Processing Unit (GPU), a Field Programmable GateArray (FPGA), USB3 controllers, storage controllers, NVMe storagedevices, PCIe device, or a network card.